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Computer vision systems provide a user a view of a scene 
whereby an image of the scene may have been augmented 
with information generated by a computer. Computer vision 
systems of the present invention include graphical user 
interfaces wbiich have been discovered to operably interact 
with geometric constructs of a user environment, objects 
within a scene, perspective of the scene, image features of a 
signal which represents the scene, among others. These 
graphical user interfaces of the invention do not behave as 
those known because operation of these interfaces depends 
on properties and feamres particular to computer vision 
systems which have position and attitude determining 
means. 
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GRAPHICAL USER INTERFACES FOR 
COMPUTER VISION SYSTEMS 

This application is a CbntiDuatioD of Ser No. 08/662^19 
filed Jun. 12, 1996 now U.S. Pat. No. 6,037,936. 

BACKGROUND OF THE INVENTION 

1. Field 

The present discovery and invention relate generally to 
graphic^ user interfaces for computer systems and relate in 
particular to graphical user interfaces for special computer 
vision systems, sometimes and herein known as "Aug- 
mented Reality™" computer vision systems. Graphical user 
interfaces of the invention find great novelty in their 
interaction, responsiveness and function related to these 
highly specialized computer vision systems. 

A clear and complete description of computer vision 
systems has been disclosed as U.S. pending patent applica- 
tion having a Ser. No. 08/119,360. In addition, some basic 
and preliminary description of graphical user interfaces as 
they may particularly relate to computer vision systems 
appears in the disclosure U.S. pending patent application 
having a Ser. No. 08/307360. Further, other concepts and 
ideas relating to graphical user interfaces, were presented in 
disclosure U.S. pending patent application having a Ser. No. 
08/411,299. Each of those three pending U.S. patent appU- 
cations is believed to contain considerably usefid informa- 
tion as it may relate to the present invention. Accordingly, 
each of those documents is incorporated herein this 
disclosure, by reference thereto. 

In agreement with provisions of 35 U.S.C. §120, this 
application is a Continuation-in-Part type application. It 
continues from Ser. No. 08/411,299, filed Mar. 27, 1995; 
which in turn continues firom Ser. No. 08/119^60, filed Sep. 
10, 1993, both remain pending as of this even date. 

2. Prior Art 

A graphical user interface is a computer generated graphi- 
cal device which a computer user may employ to interact 
with, or command, a computer system to take some action 
or actions. A commonly recognized graphical user interface 
is known quite well to most computer users as a ''Menu'*. 
One example of a Menu includes a list of option selections 
presented in a simple list box. A user may select an option 
by pointing a cursor to it via a pointing device. Some 
pointing devices include: a mouse, a trackball, and scrolling 
keys or other tactile means. Pressing "enter" or "clicking" a 
mouse button while a cursor is pointing to a selection then 
commands the computer to execute a fimction associated 
with the option selected. 

Various types of Menus have been configured to interact 
with a user in different ways. Sometimes, and depending 
upon the apphcation being run on a computer, one type of 
Menu may provide better function than another type. Two 
common types are discussed here to illustrate how a graphi- 
cal user interface may preferentially interact with a particu- 
lar application. A "pop-up" type Menu and a "drop-down" 
type Menu each act differently; eadi having certain coop- 
eration with re^)ect to the application which the computer is 
runaing. 

A "pop-up" type Menu may be initiated by some event in 
a computer program. It typically interrupts normal program 
activity. For example, if a computer error occurs, a "pop-up" 
Menu may appear in the middle of a display screen and offer 
a user the options: "continue" or "start over". 

In comparison, a "drop^down" Menu is typically initiated 
by request of a user. For example, an icon on a "tool bar" 
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may indicate a group of ta^ related to a common feature. 
Stimulating ("pointing and clicking") the icon causes a 
Menu box to drop down therefrom and into the display area. 
The Menu may have a list of possible command options 

5 which are selectable by a user. "Pop-up" type Menus, 
therefore, cooperate better with internal or automatic mecha- 
nisms which may initiate them and "drop-down" Menus 
may be better suited for functions which are initiated by a 
user. These are only a few of the many features well known 

10 in the arts of computer graphical user interface design. 

Sometimes an application which a computer is running 
suggests a certain type of graphical user interfaces. Very 
elegantly designed "drop-down" Menus having advanced 
features are used with sophisticated drawing programs. 

15 Examples which thoroughly illustrate this are the Menus 
employed by the CorelDRAW!™ drawing software pack- 
ages." Those having experience with advanced drawing soft- 
ware packages will appreciate how clever Menu configura- 
tion may greatly enhance the ease-of-use and efficiency of 

20 the application. 

There exists many fundamental differences between the 
display of a simple personal computer and the display of a 
computer vision system. A computer vision system may 
employ an electronic camera and a computer graphics 

^ generator to formulate augmented images of real scenes in 
real-time. Composite images presented at the display of a 
computer vision system may be comprised of optically 
acquired images having been modified or augmented with 
computer generated graphics. The computer generated 

^ graphics may relate to objects detected (or otherwise 
"known" by the computer) in the scene being addressed In 
particular, some objects are identified by their known loca- 
tion. The objects may be graphically simidated by, and 
superimposed onto "real" or optically acquired images of the 

■^^ objects. 

It may be desirable for the user to command a computer 
vision system to perform various functions. Standard 
Meniis, or other graphical user interfaces, can be employed 

^ by computer vision systems to provide for user interface 
fiinction. However, since computer vision systems behave 
very differently than common computer systems. Menus 
which might be most useful in computer vision systems are 
heretofore completely unknown. 

45 Particular function and features associated with computer 
vision systems which are not found in common computer 
systems suggest graphical user interfaces may be uniquely 
arranged to cooperate with those fiinctions and features 
particular to those specialized systems. The present inven- 

5Q tors have now discovered some very useful and valuable 
configurations of g^phical user interfaces as they may 
particularly apply to computer vision systems. These new 
graphical user interfaces 'provide surprisiiig results when 
considering the benefits they may provide to users of com- 

55 puter vision systems which employ them. The new graphical 
user interfaces tend to facilitate operation, enhance 
functionality, improve interpretation of images, increase 
understanding of scenes. These graphical user interfaces 
operate in a way which is not and cannot be used with prior 

^ systems. 

SUMMARY OF THE INVENTION 

A graphical user inter face system has been invente d to 
in feract with teatures and tunction which are parti cular to 
65 compute r vision s^ tgms 

Computer vision systems having augmented images may 
have a graphical user interface configured to appear to 
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interact with real objects of a scene. A graphical user graphical user interface is made to correspond at all times 

interface may be arranged to interact with the pointing with the apparent position of the object. When the pointing 

direction of the computer vision system. Graphical user direction of the vision system is panned across a horizon, 

interfaces may be responsive to position and/or attitude as objects in the scene appear on a display to move laterally. A 

determined by the computer vision system. Graphical user 5 Menu associated with a particular object can be made to 

interfaces may be re^onsive to a cursor which corresponds appear to follow the object's lateral motion so that the 

to a camera boresigfat indicator. Many configurations of graphical user interface stays with the object to which it is 

graphical user interfaces which are particular to computer attached. Even if the object leaves the field-of-view of the 

vision systems exist. These are presented in detail in the vision system, so can the drop-down Menu. When the object 

sections here following. When a computer vision system lo is again acquired by the system (comes again into the 

includes graphical user interface devices, the interaction of field-of-view), then so does the drop down Menu. Hiis 

the graphical user interfaces with respect to elements of the example illustrates that a graphical user interface may 

system can produce some surprising results. The present interact in real time with the pointing direction of the 

invention is concerned with how graphical user interfaces computer vision system, i.e. when the pointing direction is 

may be arranged to interact with computer vision systems 15 adjusted, the graphical user interface responds to the adjust- 

and elements thereof. ment. 

In addition, the absolute location of the computer vision 

Comparison to a Simple Computer system may dictate the behavior of certain graphical user 

Afundamental difference between a simple computer and interfaces. If the computer vision system is in Paris, graphi- 

a computer vision system is that displayed images in the ^ "^er mterfaces may automaUcaUy be presented m the 

computer vision system corre^nd directly to some view of ^^^"^^ language. Simflarly, if the computer vision system is 

the real world. Images are aligned to the scene in real time. determmed to be m New York, then graphical user interfaces 

A computer vision system addresses a scene as its camera automatically be presented in the English language. It 

axis is pointing toward it. The computer vision system has possible that the combmation of both position and 

associated with it at aU times, a position and attitude which ^ ^^^^^^ ^« computer vision system may affect the behav- 

are easily measurable and thus "known" to the computer. As graphical user interfaces. It is further possible that the 

a result, displayed images are presented in a certain per- ^^^^P^^y ^^^^ periphery may be enabled such that it might 

spective which corre^nds to the pointnof-view of the ^P^^^^ «° * graphical user interface. Other features and 

computer vision system and the user's position. The dis- function particular to computer vision systems can be 

played images of a simple computer are not generally ^ exploited to advance the usefulness and suggest arrange- 

assodated with its surrounding environment nor aligned to gr^hical user interfaces for computer vision 

any scene systems. The example above shows how pointing direction 

„ ., . , . . , - . might affect a graphical user interface, some further 

Besides usmg the computer vision system pomting direc- , * j j 

.,jP .-. ■ K _c !_ examples directed to position, position and attitude, magm- 

tion as an mterface pomter, a graphical user mterface may be „ u • « ^ if o -c i 

/T .1- - J- r.* 35 ncation are briefly mentioned here. Some specific examples 

arranged to respond to the pomtmg direction of the computer follow -i- r 

vision system when the bores^t is not acting as a pointer. Position 

A graphical user interface might respond to the absolute a - i i u u i • * . 

S^tin direction A sirnple example shows how position alone might dictate 

. '. . the graphical user interface content and behavior of a 

-Hie "pomtingdirecuon- of a computer vision system is a ^ graphical user interface. Since a graphical user interface may 

primary feature which ^ould be weU understood. The be an icon of arbitrary shape, it is pos^-ble that a small map 

pomtmg direction will be shown to have great interaction ^ gh^pe of a state's boundary be di^layed with images 

with various graphical user interfaces. Not only does the of scenes being addressed. As the computer vision system is 

pomtmg direction sometimes serve as an interface pointing f^^, state-to-state, the m^ could change shape to 

cursor but the absolute pointing direction may influence the 45 corre^od to the state that it is in. As a user crosses the 

behavior of certain graphical user interfaces. ^^.^er &om Kansas City, Miss, to Kansas City, Kans., a 

graphical user interfaces of the invention can be made to Missouri shaped icon would become a Kansas shaped icon, 

be responsive to a new type cursor or "pointer". Operations Position and Attitude 

known as "point-and-chck" and "drag-and-drop" were here- Under some circumstances, both position and attitude of 

tofore performed with a computer peripheral pointer known 50 a computer vision system are used to create a special version 

as a "mouse'*, "track ball", or "powerpoinr . Each of these of a graphical user interface. If a scene includes a particular 

devices allow a user to cause a pointing icon or "pointer" to object for example a billboard, then the hiUboard will appear 

Uaversc a display field. When the icon is collocated with in a different per^xiaive fd^every^location it 

something of interest in the displayed image, an object, may be viewed. It may be desirable to have a graphical user 

image or another icon for example, then some action can be 55 interface appear in the same perspective as the billboard. To 

initiated by the computer program. The pointer of a com- accomplish this, a determination of the computer vision 

puter vision system may include a conventional pointer system position and altimde enables the computer to com- 

which moves in response to a tactile stimulus, or might by pute the proper perspective associated with any place from 

arranged such that it corresponds to the vision system which the biUboard may be viewed or addressed by the 

boresight. By pointing the computer vision system in any ^ system. The graphical user interface can then be displayed in 

direction, a user causes the boresight to be collocated with a perspective which corresponds to the perspective of the 

some part of the displayed image. By pointing the computer biUboard. Thus, both position and attitude of the computer 

vision system at objects of interest, the use might indicate vision system may affect the arrangement of a graphical user 

commands to the computer. Use of a vision system boresight interface of the invention, 

as a pointer is believed to be a completely new concept. 55 Magnification 

For example, a drop-down Menu may be associated or Computer vision systems have very sophisticated zoom 

"attached" to some object of the scene. The position of the properties, graphical user interfaces of computer vision 
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systems may aid io serving zoom objectives of those sys- It is an object to provide graphical user interfaces which 

terns. A graphical user interface may be arranged such that are responsive to position or attitude, or both, of computer 

its content may include magnified images of scenes being vision systems. 

addressed, while the display field contains a non-magnified in accordance with these objectives, certain preferred 

image. 5 examples including the best modes anticipated are presented 

From the Image Signal herefoUowing in great detail with clear description having 

The electronic image signal generated by the camera may references to drawing figures, 
be probed by the computer to detect some feature. From this 

information a graphical user interface may be generated to BRIEF DESCRIPTION OF THE DRAWING 

aid in understanding and interacting with the detected fea- lO FIGURES 

FIG. 1 is an illustration of a cityscape scene addressable 

To fully appreciate the invention, one should have a by computer vision systems having various types of objects 

complete understanding of computer vision systems of the therein 

typt which produce augmented images of real scenes^ Full ^ g^,j^ i^^^, 

dBclosuie of those systems has been made, referenced and is j ^^^^ separately addressable by a computer vision 

mcorporated herem this document. A brief review follows; r j j r 

however, one cannot be expected to completely undeistand * . . ' ^ ' 

this disclosure without full understanding of the references ^ ^ it may appear to a computer vision 

as significant elements herein are defined at length in those ^5^^™ boresight mdicia therein. 

presentations. Therefore, thorough review of the incorpo- 20 ^ shows a computer vision system composite unage 

rated documents is highly recommended. having a graphical user interface therein. 

FIG. 5 show an image similar to figure four, but from a 

A Brief Review of Computer \^on System different portion of the cityscape scene. 

Concepts FIG. 6 illustrates a graphical user interface related to an 

Computer vision systems may be comprised of a camera, ^ ""^"^^ ^^^^^ ^^y2m^ an image signal, 

a computer, a display and position and attitude determining ^^slai image from a computer vision system which 

means. The system addresses a scene and produces at the * moving object with a graphical user interface 

display an augmented image thereof. The augmented image associated therewith. 

is comprised of image information from two sources. A first FIG. 8 illustrates a relationship between boresight indicia 

image source is the camera which optically acquires an ^ ^ a graphical user interface whidi has been locked to an 

image of a scene and produces an electronic image signal. image point. 

The second source is a computer generated image source. FIG. 9 shows how two graphical user interfaces might 

From position and attitude measurements, the computer look in a computer vision system image field where the 

identifies a scene and objects therein. The computer may interfaces appear to be beyond the edge of the screen, 

recall from memory, models which are related to identified FIGS. 10-12 illustrates a graphical user interface and 

objects and assembles a computer generated image having a sdieme of positioning it within an image field, 

perspective which matches the true per^jeclive of the scene p,Qs 13^14 contains an image relating to another 

- from the point-of-view of the computer vision system in real scheme of positioning I' ^aphical user interface within an 

time. Optically acquired images are combined with com- ^ image field. 

puter generated imag« to fonn composite or "augmented" pic. 15 is an image having a graphical user interface 

mages. An augmented unage is presented to a user on a responsive to a user's^osition as that position may 

display bavmg a display field abgned to the scene bemg j^late to the scene 

addressed. A user views the "real" world where the display ^ ^ , 

of the computer vision system appears to be a Wlow^ at , ^^^J computer >asion system, the apparent 

which the user looks. A user finds great benefit because the ^ ^V^ic^ user interface with respect to a scene, 

"window- may contain image information about the real ^ geometry having a relationship with the compass, 

world in true scale and proper perspective. This information ^^G* 17 similarly shows the geometry and the result of a 

may additionally contain objects which are not readily displacement in pointing direction. 

visible to the naked eye or the electronic camera as the FIG. 18 similarly shows a large displacement and the 

computer generates portions of the final image. result with respect to the gr^hical user interface. 

FIG. 19 shows a geometry which relates to a dynamically 

M P^j'^^^^f ^v..l^Vf positioned graphical user interface. ; V V ;V 

It is a primary object of the invention to provide graphical ^0 shows a geometry which includes a geometric 

interface to computer systems. 55 construct of the scene being imaged, i.e. a plane, and a 

• t L- _^ * a.- 1 - * _c * graphicaluserinterface which may be displayed to appear in 

It IS a nirther object to provide graphical user interfaces to j —r / rr 

computer vision systems. ^V. - * • , 

, . .„ ^ , . . . , . . , FIG. 21 IS an image of a object from a particular point- 

It^ still further an object to provide new graphical user ^^^^hic^ user interface appears to face 

mtertaoes. ^ normally the viewer of the image. 

It is an object of the invention to provide new fimcUon to pjQ 22 is a second image of the object of HG. 21, from 

graphical user interfaces. a different and orthogpnal point-of-view, where the graphi- 

It is an object to provide graphical user interfaces which cal user interface still appears to face normally the viewer of 

are responsive to objects and features in augmented images. the scene from that second point-of-view. 

It is an object to provide graphical user interfaces which 65 ¥\G. 23 is an image of a publicity sign promoting a 

are responsive to computer vision system functions, business service, the sign being in a perspective particular to 

features, and properties. the location from which it is being addressed. 
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FIG. 24 is a second image of ihe sign where a graphical ing but should not be used to attempt to limit the scope of 

user interface has been added to the image in a perspective any term. In other words, the definitions are formulated with 

which matches the per^)ective of the sign as it appears firom an aim to give a general understanding but not intended to 

the point-of-view of the user. set forth or limit the scope of the terms. The true and full 

FIG. 25 is a third image of the sign where a second 5 scope of each term may be determined by various means 

graphical user interface is presented as a sub-field of the first inchiding: common uses in the arts, obvious alternatives to 

graphical tiser interface. elements thereof^ similar uses in parallel arts, among others. 

nC. 26 simaarly is aji image of the sign where a second in alphabetical older and no extra importance is 

graphical user interface is presented as a sub-field of the first ""«'°**' '° ^ »° ^ 

graphical user interface. 10 Augmented Image 

. An augmented unage is a composite image compnsmg, 

HG. 27 sho>^ an miage havmg a graphical user inte^^ ^ ^ ^ ^ion and a computer 

supported by indicia which is pcsiUoned m the di^lay field ted portion. Hie optically acquired portion is gener- 

in accordance with a spatial between the graphical user ^ ^ • produced by an electronic camera. Hie com- 

mterface and the scene bemg addressed. p^^^^ generated portion is generally an image or image 

FIG. 28 similarly shows an image having a graphical user components produced in accordance with position and atti- 
interfaoe where indicia is positioned in the image in accor- tude determinations. An "augmented image" may addition- 
dance with a spatial relationship with parts of the graphical ally be comprised of graphical user interface devices, 
user interface. Computer Generated Image 

FIG. 29 illustrates a scheme whereby indicia is used in a 20 A "computer generated image" is an image or image 

graphical user interface to translate ^atial information relat- components formed by a computer image processor. The 

ing to physical properties of the computer vision system into processor may be in communication with a memory having 

a visual form. stored images therein. Additionally it may be in communi- 

FIG. 30 demonstrates a magnification mechanism relating cation with position and attitude determining means where 

to graphical user interfaces. ^ position and attitude of the computer vision system 

HG. 31 shows a "drop-and-^g" operation as it relates to ^. Vr^^' g^f ^ate various images relating to 

a zoom graphical user interface. f""^ being addressed. A a)mputer generated miage may 

„ . . - . . ... mchide graphical user mterface devices. 

FIG. 32 IS an image of a scene contaimng a graphical user Field 

interface where the content of the interface contains a 3^ "^"^Tay field" refers to a place where an image is 

magmfied image of a portion of the scene bemg addressed. ^^^^^^ ^ ^ ^^^^^^ ^.^^ ^^^^^ .^j^^ 

HGS. 33-36 contain a senes of four unages of a special- substantiaUy planar and is typically at four sides, 

ized graphical user interface which is responsive to the "Drag-and-Drop** 

display field extremities. ^ "drag-and-drop" operation refers to a procedure where 

FIG, 37 shows a graphical user interface which catalogs 35 a switch is continuously engaged while a cursor is moved 

other graphical user interfaces with respect to their position across a field, 

in relation to the user's position. Field Region 

FIG. 38 shows a similar, graphical user interface which A "field region" is a two dimensional area with arbitrary 

catalogs other graphical user interfaces with re^>ect to boundary, 

position and perspective. 40 Graphical User Interface 

FIG . 39 shows a siniilar graphical user interface having in A "graphical user interface" is a device. It generally exists 

combination therewith, a map to further detail the position as a field region in an image. It may serve to facilitate 

of certain graphical user interfaces. command of a computer or other user interface by way of 

FIG. 40 shows a scheme where the position of a graphical graphical representation of information, 

user interface with respect to "known" objects affects the Nonnally Aligned 

density of a type face used therein. "NormaUy aligned" refers to a geometric construct which 

FIG. 41 is a block diagram of a device of the invention. provides for orthoganaUty between objects which are "nor- 

maily aligned . 

FIGS. 42-44 are images having a graphical user interface Ootic Axis 

whichrespondspiDpoitionaUyinsizeoropacitytotheoflfeet ^ An "optic axis" is the symmetry axis or boresight of a lens 

between the ^mtmg direcUon and the center of the graphi- ^ ^^^^ ^ j ^ computer 

cal user mterface. 

. , vision system havmg a camera. 

FIGS. 45-47 show graphical user interlaces- wMch: ^^ - ' "^ 

respond in complexity with reflect to the pointing direction. "optically acquired image" is an electronic image 

FIGS. 48-54 show a series of images which illustrate 55 formed by a camera. Generally, a CCD type image detector 

interaction between a graphical user interface enabled com- forms an electronic image signal of a scene being addressed, 

puter vision system and a transceiver station having a It is possible that images be made from infra-red or alter- 

computer native ^ctra. In addition, "optically acquired image" may 

include images &"om sonar, radar, ultra-soimd among other 

DEFINI HONS OF CERTAIN IMPORTANT ^ ^^„^n «f ir^.iJ h.^.vU 

60 common lorms ot imagmg devices. 

"Point-and-aick" 

Certain terms and words used throughout this writing A "point-and-click" operation refers to a procedure where 

have ^cial meaning associated with them. It is important a cursor is made to be collocated with a field region while 

for perfect undeistanding of the invention that the meaning simultaneously engaging a switch, 

of these tenns be well appreciated. Accordingly, the follow- 65 Pointing Direction 

ing is presented to help further define the meaning of these Computer vision systems of the invention have associated 

terms. The descriptions should help clarify intended mean- with them a "pointing direction". Generally, a "pointing 
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direction'* is defined by and colinear with a camera lens axis opposed to a coaunon meaning: ^hisual'*. The display normal 

of symmetry. Computer vision systems having a lens nec- is generaUy aligned with the pointing direction of the 

essarily have an axis of symmetry. By aiming the camera system. Forpurposesof generality, use of the term "aligned" 

lens in various dir&ctioos, the "pointing direction" of the is clarified in detail as follows: When the display field is 

computer vision system is changed. To address a scene, one 5 perpendicular to the optic axis it is said to be aligned 

points the camera (lens axis) in the direction of the center of therewith. At times, there may cjdst an angular ofiOset 

the scene. It is possible to have a computer vision system between the display normal and the pointing direction of the 

without a camera. In this case, the "pointing direction" must camera. The di^lay is still said to "be aligned" with the 

be defined with respect to some arbitrary reference direction. camera pointing direction so long as images di^Iayed 

Position and Attitude Determining Means lO thereon are responsive to changes or di;^lacements of the 

"position and attitude determining means'* are facilities pointing direction. When the camera is pointed towards a 

which measure or otherwise determine position and attitude scene, it is said that the scene is being addressed by the 

of a computer vision system. Position may be determined system. With this arrangement, an augmented image of a 

with reference to a point on a line of the pointing direction scene being addressed may be presented in the display field, 

and within the image plane of a computer vision system 15 Images captured optically may be combined with computer 

display. Attitude may be determined with reference to the generated images to form a composite image. Computer 

pointing direction of the computer. vision system. generated images may be formed in proper perfective - 

'Real* Object based on measurements of the system position and attitude. 

A "'real' object" refers to an object within a scene As it is sometimes desirable to commaixi the computer while 

addressable by a computer vision system. Typically a "'real' 20 viewing an image of the scene, it serves objectives of the 

object" is a car, or building, etc. A "'real' object" might be system to provide graphical user interfaces, 

distinguished from an object ^ich exists in an image in an 

abstr^t sense such as "lienu" or other "image obj^". ^ SINfPLE GRAPHICAL USER INTERFACE OF 

'Real' Scene THE INVENTION 

A "'real' scene" refers to a scene which is comprised of 25 Similar to graphical user interfaces which may appear on 

real objects. It might be distinguished from a virtual scene display screens of common personal computers, it is pos- 

which may be generated in an artist's rendering but not have sible to superimpose graphical user interfaces onto the 

any correspondence with objects which might exist any- composite images of computer vision systems, 

where in space. A graphical user interface of the invention iiK:ludes a field 

Sub-Field Region 30 jegion. The field region occupies a fractional portion of the 

A "sub-field region" is a two dimensional area with display field. The field region is bounded by a periphery, 

arbitrary boundary within and enclosed by a field. some boundaries of the field region may be coincident with 

^ ^^,,™™,«,.T «^ ™^^««^^ boundaries of a di^ lay field. A graphical user interface may 

DETAILED DKO^ON OT PREFERRED ^ ^ ^ to contain information inchiding graphics and 

EMBODIMENTS 35 images. 



It will become clear that major distinction between Accordingly, the invention provides: 

graphical user interfaces of the art and those taught here can a graphical user interface for a computer vision system, 

be foimd in the fact that graphical. user interfaces of the the computer vision system having a camera with 'an»* 

invention may interact with the pointing direction of the ^ optical axis, a computer, a position and attitude deter- 

system on which they are deployed, they may interact with mining means, and a display having a display field 

portion of the system, they may interact with the perspec- normally aligned to the optical axis, said graphical user 

tive of an image being addressed, they may interact with interface being comprised of: 

objects being imaged including moving objects, they may a field region; and 

interact with particular locations in the real world, they may ^ periphery 

interact with broadcasting systems external to the computer ^ ^ ^^^^ ^ fractional portion of the 

vision system, they may mleract with abstract gpometnc g^,^ ^ j,,^ 

constructs relaUng to scen^ bemg addressed, et cetera A displaying image and graphical information while a 

sunple graphical user mterface may merely convey mfor- addressed by said computer vision 

mation to a user while more complex graphical user mter- system 

faces might provide an interaction mechanism by which a peripheries 

user might oommand a computer. . Some simple graphical user interfaces are bounded by 

ASIMPL£ COMPUTER VISION SYSimi OF "^'^^ a periphery to the field region. 

THE INVENTION Information inside the field region typically belongs to the 

55 graphical user interface and information outside is mostly 

Acomputer vision system may be comprised of: a camera, independent of the graphical user interface operation and 

a computer, position and attitude determining means, and a may belong to a more general computer application, usually 

display. The camera may have a lens with an optic axis. The the vision system. Information within the graphical user 

optic axis defines a pointing direction for the camera and interface may include, text, video, control buttons, meters, 

consequently the computer vision system. The "attitude" of 60 indicia, "transparent" fields, graphics, maps, color, desktop 

the computer vision system refers to the pointing direction objects, among others. For computer vision systems, infor- 

of the camera. The display, typically a flat panel type mation outside the graphical user interface includes aug- 

emissive display device, has a substantially planar image mented images of scenes being addressed, 

field, or simply a "display field", generally bounded on four Accordingly, the invention additionally provides: 

sides. The display field has associated therewith a "normal" 65 A graphical user interface as described above where the 

direction which is perpendicular to the plane in which it periphery is quadrilateral in shape enclosing an area 

resides. "Normal" is used here in the geometric sense as where information in the form of images and graphics 
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is displayed, said images and graphics providing an- A graphical user interface described above, said sub-field 

interface to a computer vision system user. being responsive to a cursor. 

Although most presentations herefollowing are directed graphical user interface Reference Point and Pointing Indi- 

to four-sided fields containing information therein, it is cia 

certainly possible to arrange a computer user interface with 5 Since graphical user interfaces of the invention are some- 
a complex periphery. For example, presenting a text string times used to cooperate with images having highly signifi- 
onto an image (see FIG. 19 of above dted reference Ser. No. cant spatial content, it is often advantageous to associate a 
08/307,360). A graphic c^ject like this implies a unique graphical user interface with a point of reference in an image 
periphery for every different text string. One should not i^ing di^layed. However, a graphical user interface is 
attempt to limit the graptucal mterfaces of the invention to necessarily a field region having extent in two dimensions 
wmdow-type four sided objects. Smce many of grap ^ i^^^t^l -^^ 
cal i^r interfaces m the present mvention relate to unique ^ ^ 
aspects of a computer imagmg system, there is a tendency ' - f t j ?u„ 1 <r • 
foTthe shape of the interfa^ to communicate with image ^^.^^ ^^^^^ P^"'^.- advance goal of a^- 
objects andfeatures. Several of the foUowing examples a particular unage pomt with a graphical user mterfaoe 
demonstrate this concept in further detail. 15 as a whole, graphical user mterfaces may be arranged with 
Accordingly, the invention additionally provides: » mechanism inchided to create an association. Indicia is 
A graphical user interface as described above where the ^^^^ ^o the graphical user interface m a fashion which 
periphery is of arbitrary shape and encloses an area allows the graphical user interface to correspond to a smgle 
where information is displayed, the information pro- POint; In a first example, a graphical user mterface has a 
viding an interface to a computer vision system user. 20 pointing arrow installed thereon. The tip of the arrow 
The information displayed in a graphical user interface corre^nds to the reference point. The tail of the arrow is 
field region may be static or dynamic. In the case of attached to the periphery of the graphical user interface to 
information which changes, the change is triggered by some associate the graphical user interface with the reference 
event or condition. We include as part of the computer a point. In this way, a graphical user interface comprised of a 
response mechanism which is operable for monitoring a 25 field region, a periphery and pointing indicia, may be 
condition or testing for the occurrence of an event and which associated with a single point in an augmented image, 
further drives a means which supplies the particular infor- Accordingly, the invention additionally provides: 
mation displayed in accordance with desired programming. a graphical user interface described above, additionally 
While many hundreds of possibilities may exist, it is illus- comprising pointing indicia having a reference point 
trative to mention a simple one here. A GUI with a field 30 said indicia being connected to the periphery, 
region having some information content can be made Relationsh^) of Cursor and Pointing Direction 
responsive to the camera attitude or pointing direction. it is quite important at this point to note a very unique 
When the camera points north, indicia which reflects that feature associated with computer vision systems. Images 
could be displayed. As the camera pointing direction is displayed in computer vision systems are unique to the 
adjusted, the indicia may change in response thereto. 35 pointing direction of the camera. This feature, in conjunction 
Accordingly, the invention includes: with pointing associated with graphical user interface opera- 
A graphical user interface of claim 1, additionally com- tion provides fundamental basis for many graphical user 
prising: interfaces taught here. Although "point-and-click** opera- 
a response mechanism, said response mechanism being tions are common computer command operations, the 
in communication with said field region whereby 40 "pointing" feature of a computer vision systems is unique, 
said response mechanism operates to change infor- The camera boresight may be represented by indicia in the 
mation content thereof. display field. This indicia is associated with a point defined 
Sub-fields by the intersection of a line in which the optic axis lies and 
In some versions, a graphical user interface of the inven- a plane in which the display field lies. The boresight indicia 
tion may additionally include a sub-field region contained 45 can be used in a similar way that a common cursor might be 
wholly within the field region. The sub-field region may used. The difference being that a profound relationship is 
display information independent of the information dis- established between the scene being addressed and the 
played in the field region portion that is exterior to the cursor. By manipulating the pointing direction of the camera 
sub-field region. A sub-field region might be an icon device and consequentiy the cursor (boresight indicator), graphical 
which activates a computer function when stimulated in 50 user interfaces can be related to real world objects and 
conjunction with a "point-and-click" type operation. constructs. A very important example is contrived here to 
Generally, this involves two steps. A user causes a cursor to illustrate this point further. Since computer vision systems 
■ nciove onto the position of sub-field in a first step; the user include ah attitiide determining means, the pointing direc- 
triggers the command by closing a switch while the cursor tion of the camera is known to the computer at all times. A 
remains collocated with the sub-field. Practitioners of the 55 graphical user interface can be displayed by the computer to 
arts might associate this with a "mouse clidsT. It is said that appear "fixed" to any point of the compass; for example a 
the sub-field is "responsive" to a cursor. Although it is in fact due West heading. By adjusting the camera to point West, a 
the computer which is responsive to the "point-and-click" user causes the cursor to be collocated with the graphical 
operation, practitioners of the arts might simplify the matter user interface. The relatioiKhip between the pointing direc- 
by saying the sub-field is responsive. The condition where a 60 tion of the camera and the position where the graphical user 
cursor is collocated with a sub-field while a mouse cUck is interface is displayed in the display field provides great 
engaged is tested in the computer logic processor. The opportunity for advanced graphical user interfaces which are 
sub-field is merely visual device which may appear to unique to computer vision systems. Cooperation between 
respond. images di^layed and the cursor position wfll be shown to 
Accordingly, the invention additionally provides: 65 yield great advantages to graphical user interface systems 
A graphical user interface described above, additionally taught throughout this disclosure. A user can affect a "point- 
comprising at least one sub-field region.; and, and-click" operation in a computer vision system. The 
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pointing direction of tbe camera as represented by the 
boresigbt indicia may be used as a pointing cursor. When the 
boresight indicia is within the field region of the graphical 
user interface and a switch is activated, the computer can 
detect the condition and respond by laimching a routine 
associated with the intended command. The reader will 
appreciate that although the pointing direction of the camera 
provides an excellent cursor, one that is used for most 
graphical user interface applications^ it is not a requirement 
of the invention that the boresight cursor be used. Indeed, it 
is possible to provide graphical user interfaces for computer 
vision systems where a secoixi cursor is driven by an 
alternative means such as a "trackball**, "mouse", "eye 
tradcer" or "powerpoint" device. 
Accordingly, the invention additionally provides: 
A graphical user interface described above, the optic axis 
of the camera corresponds to a point in the display field, 
preferably at its center, the computer being responsible 
to the condition when the point and sub-field are 
collocated simultaneous with activation of a switch.; 
and, 

A graphical user interface described above, wherein the 
cursor is the boresight of the camera.; and, 

A graphical user interface described above further com- 
prised of: 

a point in tbe display field defined by the intersection of 
the line in which the optic axis lies and the plane in 
which the display field lies; 

a switch having a closed position arxi an open position; 

a coincidence determining means in conununication 
with said computer, for determining if said point is 
collocated with any point in the field region and 
further for determining the position of said switch, 

whereby said computer is responsive to the coixlition of 
coincidence. 

Example having Graphical user Interfaces Responsive to 
Attimde 

graphical user interfaces may be responsive to attitude of 
the system as determined by the attitude determining means. 
As a simple illustrative example, a graphical user interface 
held fixed in position with respect to the di^lay field may 
merely display an indication of the direction on tbe compass 
in which the camera is pointing at any time. While pointing 
west, an indicator might show the text: "West**. If the camera 
is adjusted 90 degrees right, then the indicator might show: 
"North". In this way, tbe graphical user interface is respon- 
sive to the pointing direction of the system without regard 
for a boresight cursor. 
Accordingly, the invention additionally provides: 
A graphical user interface described above, said sub-field 
being responsive to the attitude of the system, 
graphical user interfaces may be Re^x)nsive to Position 

graphical user interfaces may additionally be responsive 
to position of ' the system as determined by the position 
determining means. As a simple illustrative example, a 
sub-field may merely display numerals corre^x)nding to the 
latitude and longitude of the system. This is particularly 
usefid for global applications. The reader will appreciate that 
on a smaller scale it would be possible to provide an 
alternative position unit. If one were in a warehouse, then a 
Cartesian coordinate system in feet or meters might be 
appropriate. A second simple example of a graphical user 
interface responsive to position was presented in the Sum- 
mary where a graptiical user interface having a shape 
corresponding to the border of a state is presented. 
Accordingly, tbe invention additionally provides: 
A graphical user interface described above, said sub-field 
being responsive to the position of the system. 
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The description to this point introduces simple concepts 
relating to graphical user interfaces as they might be 
arranged to cooperate with computer vision systems. Here- 
following are more examples and descriptions of advanced 
5 concepts relating to how graphical user interfaces might be 
arranged to advance further objectives of computer vision 
systems. 

Graphical user Interfaces may be Responsive to Position and 
Attitude 

10 Recall the graphical user interface having a reference 
point and pointing indicia associated with it It may be 
activated such that text di^layed therein re^)onds to the 
position of the reference point with respect to the image 
being addressed and more particularly to specific objects in 

15 the scene. While this may not appear ^>ectacular to a casual 
observer, a close look reveals its true capacity. If a computer 
vision system is addresi^ng a scene and presenting an 
augmented image of the scene at the display, then a graphi- 
cal user interface might be enabled to cooperate with objects 

20 in the scene. Objects in the scene can be "known** to the 
computer via stored information. Position and attitude deter- 
minations are used by the computer to determine exacdy 
which objects are being addressed. For example, a computer 
vision system on Alcatraz Island in San Francisco which is 

25 pointing West "knows** the scene includes the Golden Gate 
Bridge. A recorded image of the bridge can be superposi- 
tioned onto the real image of the bridge. A graphical user 
interface placed with its reference point at the North Tower 
could di^lay indication of that in the graphical user inter- 

30 face field region. 

Therefore if the computer vision system addresses a scene 
containing a known landmark, and the graphical user inter- 
face reference point is positioned to correspond to the 
landmark in the imagp, then data relating to the landmark 

35 can be displayed in the graphical user interface sub-field. 
Merely causing the reference point of the graphical user 
interface to be coincident with a different landmark (moving 
the graphical user interface) would cause the sub-field to be 
updated with new information which relates to the new 

40 landmark. 

Graphical user Interface may be Re^)onsive to Image Signal 
The computer can analyze the image signal provided by 
the camera in a pixel-by-pixel fashion. 

Consider a graphical user interface having a text field and 
45 additionally having a reference point associated therewith. If 
the point corresponds to an image pixel which is red in color, 
then the text field may indicate the color of the image at that 
point. If tbe graphical user interface is moved to another 
location in the image, then the reference point would be 
50 associated with a new pixel, perhaps having a different color. 
The graphical user interface text field could display new text 
corresponding to the new color. This example is simple and 
tends to want utility; however, ir illustrates a powerful point. 
The graphical user interface may interact with ttie image 
55 signal in real time. 

Accordingly, the invention additionally provides: 
A graphical user interface descnbed above, said graphical 
user interface being responsive to an image signal 
generated by the camera. 
60 It is certain that one will now gain strong appreciation for 
the true utility of such a novel arrangement. Tbe clever 
combination of graphical user interfaces with computer 
vision systems, and particular features thereof including the 
attitude (pointing direction), position, position and attitude, 
65 among others; yields surpri^g results. In order to provide 
a thorough and complete disclosure, the following embodi- 
ments are presented with reference to drawing figures. 
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\^^tb reference to drawing FIG. 1, a dtyscape scene is of 
interest to illustrate some graphical user interface devices. 
The scene includes particular types of entities or objects, 
lliese include: a) mobile or moving objects including: a boat 
1, cars 2, and clouds or airplanes passing in the sky; b) fixed 5 
or stationary objects including: buildings 3, land, and a body 
of water 5; and c) a third type may include semi-stationary/ 
mobile (Ejects such as a crane 4 which tend to be stationary 
like a building but may sometimes move. A computer vision 
system may address various portions of the entire cityscape lO 
at different times. It will be use^ for later presented 
examples to divide the cityscq)e of FIG. 1 into three 
individual scenes, 21, 22, and 23 of FIG. 2, each represent- 
ing a single scene independently addressable by a computer 
vision system. FIG. 3 shows the scene 21 as a single image is 
as it may be presented in a computer vision system. A 
graphic representation of the camera boresight is shown as 
31 in the center of the image. If the computer vision system 
is parmed to the right, the boresight mark remains in the 
center of the di^lay; however, the images appear to move 20 
left in the normal manner which may be observed in 
common electronic cameras. If a gr^hical user interface is 
initiated in the presented image as ^own in FIG. 4, it may 
include a field region 41, arrow indicia 42, a reference point 
43, and sub-fielck 44. The sub-fields 44 may contain data 25 
which relates particularly to the building associated with the 
graphical user interface by way of its relationship 
(superposition) to the reference point 43. Since the computer 
"knows^ the position and attitude of the computer vision 
system, it is determined that the Opera House must appear 30 
at the boresight of the image. Accordingly, the computer 
presents in the graphical user interface sub-fields informa- 
tion whidi relates to the Opera House. This information may 
have been previously recorded in a computer memory. 

Other objects in the cityscape may also be of interest. 35 
When the computer vision system is parmed right to view 
other buildings in the skyline, new graphical user interfaces 
may be requested by the user. FIG. 5 shows another building 
51 known to the computer via its location with respect to the 
computer vision system as determined in position and atti- 40 
tude measurements. The grq)bical user interface 52 has 
information in the sub-fields 53 which is different from that 
presented in 44. The new information is particular to the 
building associated with the new graphical user interface 
reference point 54. 45 

Some temporary objects may be unknown to the computer 
due to its limited data base. For example, FIG. 6 shows a 
crane 61 in the skyhne which may have been erected after 
the time when the computer was last programmed with data. 
In ttus case, the ^^mknown'* object may be detected in the 50 
image signal but no information can be recalled. A sub-field 
62 reflects this condition. . A graphical user interface being 
initiated for'this object would have' limited data to present. 
It could present in sub-fields 63 information relating to the 
cranes height, color, range from the user, etc. 55 

Moving objects may be tracked by the computer. With 
knowledge of range and angular differencing, the computer 
may compute information about the moving objects. FIG. 7 
shows a sailboat 71 where a graphical user interface 72 has 
been initiated arK) associated with the moving object. Again, 60 
it is unlikely that a computer database could know details 
related to the object. However, sub-fields 73 might display 
information relating to the object or properties of the object 
such as ^ed and color. 

The examples show bow graphical user interfaces may be 65 
associated with a particular object of a scene via its 
co-location in the display field. In prior examples, a graphi- 
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cal user interface was initiated for objects at the point of the 
boresight The graphical user interface reference point and 
the boresight were shown to be collocated. It is possible to 
"release" the graphical user interface from the boresight and 
leave it at an object such that it remains associated with that 
object. Upon a "release" command, the computer remem- 
bers where the graphical user interface is located and forces 
it to remain associated with that point in the image regard- 
less if further adjustment to the camera pointing direction is 
made. The graphical user interface of FIG. 4 being initiated 
and then released, remains associated with the Opera House 
even if the camera is panned away so that the boresight no 
longer corre^)ODds to the reference point of the graphical 
user interface. FIG. 8 shows an image of a scene having a 
graphical user interface 81 associated with the Opera House, 
with sub-fields 82 having data particular to the Opera House, 
and a reference point 83 located in a position of the image 
not associated with the boresight 84. There are several 
engineering solutions to maintaining the association of a 
graphical user interface with re^)ect to a point in the real 
world. A first is to assign the graphical user interface 
reference point a unique position and attitude value upon a 
release command. A second may include a scheme to probe 
the image signal and apply pattern recognition techniques. 
Regardless of the particular mechanism which may be used, 
a graphical user interface held fixed in relation to an image 
in a computer vision system is believed to be quite new. 

Since graphical user interfaces may be "left" at some 
location in the cityscape, many of them may be initiated at 
various points therethroughout. They may even disappear 
from the field-of-view of the computer vision system at any 
given time, but still remain associated with the object to 
which they were assigned. When the computer vision system 
re-acquires the scene containing the object, then the graphi- 
cal user interface reappears. FIG. 9 shows portions of two 
graphical user interfaces, 91 and 92, placed on objects and 
"locked" thereto. In this example the display field boimd- 
aries are considered to partly make up the graphical user 
interface periphery. Note that a portion of the graphical user 
interface seems to extend beyond those limits. 

One might note a slight difference between graphical user 
interface 91 and graphical user interface 92. The arrow 
indicia is located in the opposite lower comers. It is usefiU 
to allow the position of the point of reference to be selectable 
by the user. Since an image may contain a certain portion 
which preferably should not be blocked by a graphical user 
interface, graphical user interfaces are allowed to have an 
adjustable reference point and pointing indicia. After a 
graphical user interface has been initiated and released from 
the boresight, it may be manipulated further. Activation of a 
particular computer routine may cause the gr^hical user 
interface to be translated along an arc centered at its refer- 
ence point. For example, the' boresight can be iised to "grab"^ 
the graphical user interface at any point in its field region. 
Further adjustment of the camera pointing direction then 
causes the graphical user interface to move along the arc 
centered at the reference point FIGS. 10-12 show how this 
might look. A reference point 101 has an arc 102 associated 
with it. Arrow indicia 105 "attached" at the graphical user 
interface periphery would be slidably movable along the 
entire periphery 104 while its tip remains fixed at a point in 
the image. FIG. U shows that advance of the pointing 
direction about the arc 112 would cause the graphical user 
interface to be moved to a new position in the image while 
keeping its reference point HI stationary with respect to an 
object in the image. During this process, the arrow tail 113 
remains attached to the graphical user interface at its periph- 
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ery 114 where it appears to slide tbeiealong. Finally, FIG. 12 16 where the pointing direction of the computer vision 

s1k>ws the completed operation with the graphical user system has been slightly adjusted. The camera pointing 

interface in a new location, the arrow tail 123 attached to the direction 174 no longer points West. As the field-of-vicw 

periphery in a new position, and the reference point 121 172 moves right (it necessarily moves with the pointing 

remaining at the same place with respect to the Opera 5 direction), the graphical user interface 173 appears to move 

House. While this rotation about a fixed point is useful, it is to the left of the field-of-view as it remains on a due westerly 

noted that there are other possible schemes of moving a heading as maintained by the computer. la the event that the 

graphical user interface about an image while maintaining a computer vision system is pointed North, then field-of-view 

relationship with an object in a scene being addressed. One would no longer contain the graphical user interface. The 

might note that the Opera House actuaUy occupies an area lO graphical user interface would ''disappear^ and not be dis- 

of image. Indeed, there is nothing significant about the exact played in the display field. FIG. 18 shows a field-of-view 

point chosen for the examples in FIGS. 10-12. FIG. 13 181 which corresponds to a camera 182 pointing 183 North, 

shows an outline 131 which contains the image area occu- The graphical user interface 184 remains off in the westeily 

pied by the building of interest. Using a "drag-and-drop" direction 185 in relation to the user. Even though the 

operation, one may move the graphical user interface to a 15 graphical user interface is not di^layed, the computer 

new location in the image shown in FIG. 14 while main- remembers where the graphical user interface is. Should the 

taining the relationship of the graphical user interface and user return the pointing direction to the West; the graphical 

the Opera House. So long as the graphical user interface user interface would be re-acquired. It is now clear that 

reference point is associated with any point on the building, graphical user interfaces of computer vision systems might 

then information di^layed therein could be arranged to 20 be arranged to cooperate with geometric or topologic fea- 

correspond with that building. It is duly noted here that tures of a user's environment Of course, since simple 

c^gg^g dropping a graphical user interface is a com- personal computers typically do not have dynamically mov- 

mon operation in some computer programs, however when able display screens or "pointing directions", this coopcra- 

the camera boresight is used as a cursor, the resulting images tion between graphical user interfaces and the pointing 

behave in a greatly different fashion providing a very useful 25 direction has heretofore been completely unknown, 

and unique effect. More complex relationships also exist For example, a 

A graphical user interface may be moved automatically combination of the direction to a stationary object and the 

without manipulation by the user. A first example relates to instant pointing direction may provide basis for locating a 

a moving object which can be tracked by probing the camera graphical user interface with respect to a field-of-view. FIG. 

image signal and detecting motion. If a graphical user 30 19 shows a camera 191 of a computer vision system where 

interface is "fixed" to a moving object, then the graphical the camera has associated therewith an attitude indicated by 

user interface would move while the object moves. This was direction 192. A known object, or in the present case a 

first shown in FIG. 7. Asecond example is illustrated in FIG. building 193, has a direction 194 defied by a vector between 

15. The image iachides a scene of a river 151 aiKl mountain the camera and the building. A graphical user interface 195 

range 152. A trail is featured in the augmented image as a 3S may be made to appear in the direction 196 which bisects the 

highlighted path 153. Agraphical user interface 154 could be angle Phi 197 between those directions. If the camera 

made to follow the path while remaining at a predetermined pointing direction changes by an amount DeltaPhi, then the 

apparent size. As the user advances along the path, the graphical user interface changes by an amount equal to half 

graphic^ user interface always appears to subtend the same DeltaPhi. The graphical user interface would appear to 

solid angle; the graphical user interface appears to follow the 40 "float" as the pointing direction is varied, 

path ahead of the user. It is an expert reader who will recognize a curious 

The previous examples show quite vividly how graphical phenomena at this point The graphical user interface 195 

user interfaces of the invention might interact with objects tends to appear without perspective. At all times, it is 

being imaged by a computer vision system. The following normally aligned with respect to the direction 196. However, 

presents a detailed look into relationships graphical user 45 this is not a requirement. FIG. 20 ^ows a construction 

interfaces might have with a computer vision system user's having a building 201, a camera 202, a camera pointing 

enviromnent in general. For the following discussion, it will direction 203, and a proposed rotation of the camera away 

be advantageous for the reader to recall concepts of geom- from the horizon in an upwardly direction 204. A graphical 

etry as they may relate to a user's point-of-view. For user interface may be defined to appear at all times perpen- 

example, the directions of the compass North, South, East 50 dicular to the horizon while simultaneously being at the 

and West. Compared to the prior section, compass direction boresight of the camera such that the graphical user interface 

is totally independent of scenes being addressed and any always appears to be in the plane 206 showti. By pointing the 

subject matter therein? AdditioDally,' the notions of "a plane camera straight up, or 90 degrees £roin the horizon, Vuser^"* 

parallel" or "a plane askew" with respect to the camera would cause the graphical user interface to move off to an 

pointing direction is unaffected by any subject or scene 55 infinitely far away position as it would be viewed from the 

being addressed. Wilh that background note nicely set, the user's location. This shows one of the many possible 

following wUl then receive due attention. arrangements where graphical user interfaces are arranged to 

Wi)h reference to FIG. 16 where a compass 161 is interact with geometric constructs or topological features of 

iUustrated along with a computer vision system 162 having the user's surroundings. In most apphcations, it is antid- 

a pointing direction 163 and field-of-view 164. Within the 60 pated that a gr^hical user interface will appear normal to 

field-of-view a graphical user interface field region 165 may the camera pointing direction. This is illustrated in FIGS. 21 

appear. The computer may be programmed to keep the and22 which show a single object being addressed from two 

graphical user interface on a westerly heading regardless of directions orthogonal to each other. The same graphical user 

the pointing direction of the camera. For example, if the interface is presented in each image where its orientation 

camera is panned slightly right (North), then the graphical 65 with re^ct to the object appears to be rotated. The graphi- 

user interface appears to move to the left side of the cal user interface is made to appear perpendicular to the 

field-of-view. FIG. 17 is a drawing similar to the one of FIG. camera pointing direction firom both poinls-of-view. 
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It is DOW easily appreciated that graphical user interfaces 
may be arranged to cooperate with the pointing direction of 
a camera in a fashion which has heretofore been completely 
unknown. The sections above illustrate some spectacular 
efifects which can be achieved with graphical user interfaces 5 
of the invention. Still further relationships between graphi- 
cal user interfaces and images produced by computer vision 
systems exist. These are not necessarily based upon the 
position of an object as was demonstrated in the first section 
presented, nor the topologic and geometric configurations of lO 
the user's environment as shown in the second section, but 
they are related more closely to the perspective of images 
being addressed. The following sections introduce addi- 
tional novel relationships between graphical user interfaces 
and computer vision systems. 15 

Most common uses of graphical user interfaces relate to 
menu selections. There is generally no value nor function 
associated with the shape of the device. Consequently, 
common graphical user interfaces are typically rectangular 
in shape. Since graphical user interfaces of the invention can 20 
sometimes be related to a real c^ject in a scene being 
addressed, and since objects in a scene are generally three 
dimensional, graphical user interfaces might preferably have 
attributes of 3-D objects; specifically, perspective. When a 
computer vision system forms an image of a real object, it 25 
appears as a two dimensional shape having a certain per- 
spective. FIG. 23 shows a publicity billboard for a boat 
cruise service company. The actual sign 231 is rectangular in 
shape. However, ^nce the image was captured from a 
certain point-of-view, below and left of the sign, the sign 30 
appears in perspective. In the image plane the boundary of 
the sign forms a quadrilateral shape. Similarly, the text 232 
appears in a perspective unique to the point-of-view. For 
every location from which an image of the billboard may be 
made, there exists a unique perspective and hence shape in 35 
which the sign will appear in a 2-D image. 

Where graphical user interfaces are made to cooperate 
with such objects, it inay be advantageous for them to appear 
in a corresponding perspective. For example, if the function 
of a certain graphical user interface is to translate the written 40 
language of the sign into a language understood by the 
computer vision system user, then it would be useful to have 
the graphical user interface to appear in an identical per- 
spective with new words. FIG. 24 shows a graphical user 
interface 241 having a periphery whidi is quadrilateral in 45 
shape and which exactly matches the shape of the real 
billboard as it appears from the point-of-view of the com- 
puter vision system. Similarly the translation text 242 and 
243 has been presented in proper perspective. Billboards 
have the simple function of relaying information to a reader 50 
graphical user interfaces of the invention may and addition- 
ally provide for user interaction. Iii the presented example, 
certain sub-fields 244 have been included to illustrate facili- ^ 
ties for user interaction with a graphical user interface 
configured as a virtual billboard. In this way, a user might 55 
now interact with the sign by '^cUcking'* on a sub-field of 
interest The sub-field 251 in FIG. 25 having the German 
word "KURS" therein can be activated to reveal a new 
graphical user interface 252. Since the image displayed in 
graphical user interface 252 is a map which does not 60 
correspond to any object in the scene., it may be best 
presented without perspective. The course 253 can be shown 
in the map which indicates where the boat cruise will go. A 
sub-field 2£2 of FIG. 26 relates to a second function. By 
"clicking^ on that sub-field, a user can learn about the cost 65 
of the trip including a mechanism to convert between 
German and American currencies. Since the graphical user 
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interface 261 does not directly relate to any object in the 
scene, it is presented in a simple rectangular shape. When a 
computer vision system is moved to a new location, objects 
of the scene appear to take a new shapes (perspective is 
different for every point-of-view). A graphical user interface 
which takes the shape of an object in perspective, can 
similarly change its shape for every point-of-view. This is 
possible because the computer is constantly updated with 
position and attitude information which can be used to 
compute the perspective of any object "known" to the 
computer. 
The invention provides: 

A graphical user interface as described above, where the 
periphery corresponds in shape to a three dimensional 
object in the scene as it appears in perspective from the 
point-of-view of the computer vision system. 
Although a billboard illustrates an object in per^)ective 
quite well, it does not account for the effects which may be 
observed when considering objects having appreciable 
depth. For example, a billboard having a cubic shape having 
six sides. When one addresses such billboard from various 
per^ctives each of the sides may come into view. In this 
way, we may provide graphical user interface which appears 
as e-D objects having depth. 

Similar to simple graphical user interfaces of the 
invention, graphical user interfaces which appear in proper 
per^ctive may incorporate sub-fields for various purposes. 
The sub-fields may have simple rectangular shape or may be 
of complex shape and may correspond to the perspective of 
some object. They can be arranged to operate as "push 
buttons" where they are responsive to a cursor and "point- 
and-chck" operations. 
Therefore the invention also provides: 
A graphical user interface as described above, where the 
field region contains a plurality of sub-fields, each 
contained entirely within the periphery of said graphi- 
cal user interface, 
and additionally: 

A graphical user interface as described above, where 
the plurality of sub-fields comprises at least one 
sub-field >^ch oorre^nds in shape to a three 
dimensional object in the scene being addressed as it 
appears in perspective from the point-of-view of the 
computer vision system, 
graphical user interfaces may contain information which is 
not as useful when presented in perspective. However the 
information may still relate to physical a^cts of the scene 
being addressed This is the case for graphical user inter- 
faces which may contain a map. Since map information 
displayed in the graphical user interface is of a nature where 
it is best understood without any distortion due to perspec- 
tive. A map is best presented without perspective, as per- 
■^ctivc causes a varying scale over the area of the map. Buf- 
a map might be more useful if one is able to see how it 
relates to a scene in the perspective firom where the user 
might view the scene. For clarity, it is possible to generate 
a graphical user interface in a rectangular shape and com- 
bine it with certain indicia to provide a oonformal mapping 
scheme which indicates where the map boundaries would 
appear in the scene. Accordingly, a graphical user interface 
can be arranged to display a map in a rectangular shaped 
graphical user interface as in FIG. 27. That map graphical 
user interface 273 can be then be combined with indicia in 
the form of a lines 271 which represents the four sides of the 
map. Oonformal translation results in indicia having a shape 
which contains six sides including two sides of the display 
field. Parts of the indicia may be "hidden" behind the 
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graphical user interface itself depending upon where the graphical user interfaces are not only dynamic in shape as 

graphical user interface is positioned in the display field presented, but they may be dynamic in size as well. So 

Graphical user Interfaces of the Invention therefore include: long as the graphical user interface field region is a 

Graphical user interfeces as described above, additionally subset of the dispUy field, then it may be of any size 

comprising indicia in the shape of a polygon having at ^ without practical restriction. RG. 30 shows a small 

least three sides, each side of the polygon correspond- graphical user interface 301 containing the text letter 

ing a side of the periphery of the field region or the "t'i an expanded graphical user interface 302 

boundary of the di^lay field, the periphery of the field having the same letter 303. The large graphical user 

region corresponding to some physical feature of the interface occupies a larger portion of the display field 

scene being addressed as it appears fi^m the point-of- lo blocks more of the image of the scene. Certain 

view of the computer vision system. applications may find use in allowing a single graphical 

It is useful to show a translation guide between the user interface to switch fi-om a small aze to a large size 

rectangular periphery of the graphical user interface and the and back again. Maximum detail in a scene can be 

indicia lines which represents the rectangle in perspective. realized when the graphical user interface is small. If 

By presenting indicia 272 which connect the comers of the 15 interaction with the graphical user interface is desired, 

recUngle with the appropriate points in the scene it is it can be "called". When it is called, it can be 

visually easy to make the translation between the map and expanded to facilitate interaction therewith. After use, - 

the real world. it ^an be "sent back" to its small iK)n-interfering size. 

Therefore: "Call** and "send back** zoom type functions can be 

A graphical user interface as described above, addition- ^ , computer imap generator routines 

aUy comprising indicia in the form of translation lines ^ addUional type of graphical user mterf ace involves 

which provide visual translation by connection another magmfit^n feature. A graphical user interface is 

between corresponding points on the periphery of the ^ P^"^'' tT^'^ . Tk" 

graphical user interface and points on the indicia poly- '^"18 l^'f^ f'J^i ""^^ 

gon which are related to each other. ^5 unage by way of ? "cUck and drag operation to define a 

is provided region to be magnified. This is illustrated m FIG. 31 where 

Map graphical user interfaces described have an area that » °f » "'y«=>P« jl ^ "^'t 

is to be translated into proper perspective with respect to the fx^^^ou^ > start pomt 312. While holding a button, the 

image of the scene. It is not always an area that is desirable !f dragged m the dirertion of the arrow to a stop pomt 

to be translated, but sometimes a line, path or route. HG. 28 3" where the button is released ■ J^ie area selected 

shows a map with a route 281 in one perspective (i.e. indicates the unage region to be magmfied. In respon^ to 

generally from above), but the same route appears in the "^"^ defimUon of the area, the computer provKtes a new fieU 

scene in a different perspective. Indida 282 in combination showmg a rnagmfied image therem. FIG. 32 shows the 

with the route of a graphical user interface presents a means 5*^' °J I ^'^^ °? .'^^^ f 

to understand how a route may appear to a user from any defined by chck and drag" for a magmfication operaUon. A 

point-of view second field 322 is provided with a magmned image of the 

Tlie invention inchides: area therein. - ^ u - - 

,. , ^ J -L J L J J-.. ,1 Many examples of graphical user mterfaces havmg mter- 

a graphical user interface described above, addiUonally ^^-^^ -^^^ ^^^^^^ ^^^^ presented/*'^ 

cornpnsmg mdicia forming at least two paths, each ^ ^ ^^^^^^ hru^al element of the computer vision 

path bemg composed of a series of pomts, ^^^^^ ^^^^ ^ ^ ^^^^^ ^^^^^ ^ 

a first path being within the field region; and graphical user interface. The display field boundaries 

a second path being within the di^lay field, the second might be activated to act upon a graphical user interface sudi 

path oorrc^nding to the first path, the second path that it influences its position within the field-of-view. This 

being shown in a per^)ective which has been translated 45 specialized type of graphical user interface includes one 

from the perspective of the fiirst path to a per^ctive having an association with a particular point in the image 

which corresponds to that of the scene being addressed. until displacement of the pointing direction causes a portion 

Hie example of FIG. 27 presents a case where an image of the graphical user interface periphery to be collocated 

is modified to show the map boundaries. Similarly, it is with the boundary of the display field. When a portion of the 

possible to modify the map to show the computer vision 50 periphery is collocated with the edge of the display field, the 

system*s boundaries. The field-of-view for a camera is a graphical user interface would be released from its fixed 

cone. In two dimensions, it takes the shape of a triangle. position and advanced across the scene until fiirther dis- 
FIG. 29 shows indicia 291 displayed on -the m^ of 'the placement of the pointing direction stops. At that time, the 

graphical user interface to indicate the extent of the field- graphical user interface is "dropped" or associated with a 

of-view of the camera. Of course, if the lens is caused to 55 new point in the image and remains fixed thereto until the 

perform a zoom function, then the apex angle of the graphic display field and graphical user interface edges are again 

would be re^nsive to that change. Since a cone which collocated. In this way, a graphical user interface is made to 

represents field-of-view extends infinitely far, the graphic remain continuously within the display field regardless of 

must be limited at the map boundary. Indicia which repre- the extent of change in the pointing direction, 

sents the field-of-view in a map graphical user interface is FIG. 33 illustrates a scene of the Great Wall 331 of China 

typically a four-sided or three-sided polygon. and a graphical user interface 332 containing text informa- 

Accordingly, tion: "Great Wall**. The graphical user interface may be fixed 

a graphical user interface as described above, additionally to a point 333 associated with an object in the scene being 

comprising a sub-field region having at least three addressed. Displacement of the pointing direction to the 

sides, two of which corTe^x)nd to the boundary of the 65 user's left causes objects and consequently the graphical 

field-of-view of the camera, user interface to appear to shift right. FIG. 34 shows an 

is provided. image where the pointing direction of the computer vision 
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system has been shifted a few degrees causing the graphical played in proper perspective with respect to the scene 
user interface 341 to move closer to the center of the display graphical, user interface 383 is a "radar^ graphical user 
field as it remains fixed to the point associated with the real interface which imdks and catalogs any graphical user 
object 342. At this location, the graphical user interface interfaces launched and in use. Location and perspective can 
edges are far from the display field edges. However, if a 5 be represented. Line segment indicia 384 represents the 
large displacement were made the edges would finally normally aligned graphical user interface 381 and line 
become collocated therewith. FIG. 35 shows an image segment indicia 385 represents graphical user interface 382 
where a large displacement to the right has caused the edge displayed in perspective. In some cases, it is additionally 
of the display field 351 to "acquire** the graphical user usefid to supplement the radar graphical user interface with 
interface 352 and "unlock** it from the point (no longer lo a map baclqground. FIG. 39 shows a radar graphical user 
shown) which it was associated. Continued displacement to interface having a map background, 
the right causes the graphical user interface to move with A gr^^hical user interface can be arranged to respond to 
respect to the scene until the pointing direction displacement objects in a scene to proximity to an object of concern. To 
is changed again to the left. Adi^lacement to the left causes indicate a degree of nearness, the graphical user interface 
the graphical user interface to become "locked" to a new is text can become more bold or faint in proportion to how near 
point in the scene. FIG. 36 shows an image where the the graphical user interface is to the object it is associated 
graphical user interface 361 no longer foUoi^'s the edge, but with. In addition, it may change its content entirely as it 
is "locked" to a new point 362 in the scene. This type of approaches new objects. FIG. 40 shows a landscape scene 
graphical user interface is particularly useful when a user containing a known mountain range. A graphical user inter- 
wants the gr^bical user interface to always be present in the 20 face 401 having been positioned such that its pointing 
display field but does not want it at the boresight. indicia is at "Canary Peak" has text in a bold typeface. If the 
Now, many types of graphical user interfaces have been graphical user interface is repositioned by any means, to a 
described including graphical user interfaces which relate to location near, but not directly at Canary Peak, then the text 
objects of a scene being addressed, graphical user interface in the graphical user interface 402 begins to fade to a less 
which interact with geometric constructs of a surrouiiding, 25 dense type. The further the graphical user interface is from 
graphical user interfaces which provide magnification the known point in the image, the lighter the text, graphical 
sdiemes, graphical user interfaces which interact with dis- user interface 403 has text which is barely readable thereby 
play field extremities et cetera. A still further new type of reflecting its position far from Canary Peak. Continued 
graphical user interface can be provided. A graphical user displacement causes the graphical user interface to become 
interface which operates to catalog and track other graphical 30 nearer a second known point. As the graphical user interface 
user interfaces being used in a computer vision system. crosses the midpoint between Canary Peak and Owl 
These specialized graphical user interfaces may contain Moimtain, the content of the graphical user interface text 
positional information about other graphical user interfaces. changes to reflect that, graphical user interface 404 shows 
For example, if a scene contains a plirrahty of graphical user "Owl Mountain" in Ug^t text to indicate that the graphical 
interfaces distributed throughout space, it may be useful to 35 user interface is far from, but approaching Owl Mountain, 
catalog the graphical user interface in a positional diagram. As the graphical user interface is repositioned closer to the 
A diagram may be created where the system user's position known point, the new text is presented in increasingly darker 
corresponds to the center of the diagram. The position of type (graphical user interface 405 and 406). Finally, when 
each graphical user interface can then be shown in relation the graphical user interface 407 position corresponds 
thereto. In addition, for graphical user interfaces not pres- 40 directly with the exact position of the known point, the 
ently within the field-of-view but having been previously darkest possible text is di^layed. 

"placed" in some location, the diagram could indicate their A similar but unique graphical user interface may have a 

position with respect to the field-of-view in order to facili- behavior which is proportionally responsive to the degree to 

tate re-acquiring those graphical user interfaces. FIG. 37 which the system is pointing to a particular object. FIG. 42 

shows an image having four graphical user interfaces 45 shows an image of San Francisco including the famous 

therein. A first graphical user interface 371 having pointing TransAmerica building. A graphical user interface 421 

indicia marks a particular location in the scene. A second affixed to the building can be made to disappear slowly in 

graphical user interface 372 is "attached" to an object (Owl proportion to the ofiEset between the system pointing diiec- 

Mountain), the graphical user interface being merely a tion and the direction to the building. For example, as the 

location label. A third graphical user interface 373 contains 50 vision system camera is parmed away, the graphical user 

continuously updated date and time and is responsive to interface is made more transparent allowing the scene 

environmental conditions and information which is particu- behind it.to show through. FIG. 43 shows an image inchid- 
lar to the instant computer vision system such as: position, * ' *" uig graphical user interface 431 which has faded to reflect 

pointing direction, and temperature. Their positions with the pointing direction does not correspond to the direction of 

respect to the user might be graphically represented in a 55 theobject/graphicaluserinterface. A nearly identical mecha- 

fourth graphical user interface 374 graphical user interface nism could cause the graphical user interface to shrink in 

374 having a circular periphery also has two radial lines size. RG. 44 shows graphical user interface 441 which 

which indicate the extent of the field-of-view. The center of appears smaller than 421 to reflect the condition of the 

the graphical user interface represents the user's position. displacement of the pointing direction away from the Tran- 

Short line segments distribute about the graphical user 60 sAmerica building. A similar system allows the detail of 

interface correspond to the graphical user interfaces pres- information presented in a graphical user interface to vary as 

ently addressable by the computer vision system. Three are the pointing direction corresponds more closely with an 

at least partially within the field-of-view and three others are object. FIG. 45 shows a graphical user interface 451 having 

outside the field-of-view. considerable detail including: a map 452 which moves in 

The drawing of FIG. 38 shows a graphical user interface 65 response to changes in user's position and attitude, a video 

381 being normally aligned with respect to the viewing field 453 which shows the inside of the building, and push 

direction and a second graphical user interface 382 dis- button sub-fields 454 which can be activated to find more 
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informatioa/interactioDS. Whea the pointing direction is interface 523 to for confirmation of the booking graphi- 
panned away from the TransAmerica building, then it is cal user interface 524 provides further instruction by way of 
anticipated that less interaction is desired. Therefore, the text. Confirmation by clicking on the graphical user inter- 
graphical user interface responds by displaying less infor- face 523 allows the computer at the building to be manipu- 
mation. FIG. 46 shows an image where the graphical user 5 lated by a computer vision system user. The user may 
interface 461 has been reduced in complexity and detail in approach other institutions operating transceiver stations 
response to the new pointing direction which corresponds to with the necessary protocol to interact with those business 
other parts of the cityscape. Finally, FIG. 47 shows that a with the same computer vision system, 
large displacement causes the graphical user interface 471 to FIG. 53 shows graphical user interface 531 having three 
be reduced to its simplest form. lO sub-fields 532 which corre^>ond to accounting information. 

Some specialized versions of graphical user interfaces of This information is used by the transceiver station to receive 

the invention are interactive with transceiver systems. A money transfer in response to user interaction with graphical 

computer vision system may be arranged to receive electro- user interfaces of the invention, graphical user interface 532 

magnetic signals (radio, IR, etc.) from transmission facilities provides confirmation of the purchase, and further instruc- 

and to return signals thereto. By way of this type of link-up, 15 tions. FIG. 54 shows a final image having a graphical user 

a computer vision system can communicate with systems interface 542 which provides similar summary remarks 

which may be remotely located. Data and information may along with additional pleasantries. 

be exchanged between the computer vision system and the Although the combination of the transceiver station and 

transceiver system in a way to advance simple objectives computer vision system may be envisaged as a single device 

such as purchases of services including scheduling an 20 for purposes of defining an invention, the computer vision 

appointment. The interface between the himian user and the system having predefined protocol and being enabled with 

computer vision system/transceiver system can be provided graphical user interface capability is also considered a 

as a graphical user interface particular to computer vision complete device as it stands alone. 

systems. An example of such arrangement follows. It has been clearly shown that graphical user interfaces 

The image of FIG. 48 is a part of a facade of a service 25 can be arranged to be responsive to position and attitude 

provider business. The facade is stationary as it is part of a determination in many ways. It is important to understand 

building. The location may house a computer with a broad- that the few examples presented here are only representative 

casting and receiving facilities. The transceiver computer of the many thousands of ways to incorporate position and 

may be configured to conununicate as a complement with a attitude response mechanisms into various graphical user 

plurality of computer vision systems via a communication 30 interface devices. What is con^dered to be the essence of the 

protocol. Similarly, a computer vision system may visit invention is the graphical user interface for a computer 

various facades which may communicate with it. When the vision system. 

facade is addressed by a computer vision system, graphical FIG. 41 shows a block diagram of some system elements 

user interfaces provide a facility for user interaction there- as they relate to each other Particularly, a display field 410 

with. This may include many various types of interaction, 35 is a planar region operable for producing thereon a light 

one example is illustrated here. emissive pattern. Afield region 411 is a fractional portion of 

FIG. 49 shows how the facade may look to an English the display field which contains image information therein, 

speaking computer vision system user. Signs originally A sub-field region 412 is a fractional part of the field region 

written in the Chinese language have been translated into the and may similarly contain image information therein. A 

English language words "Acupuncture" ai^ "'Massage'' in 40 camera 413 transmits an electronic image to a computer 414 

the graphical user interfaces 491. In addition, a salutation having an image processor 415. EV>sition and attitude deter- 

inchiding instructions is provided via graphical user inter- mining means 416 and 417 re^)ectively produce a signal 

face 492. By cliddng on either graphical user interface 491, which drives a computer memory 418 having output to the 

a response is affected which provide for additional input. image processor. Position and Attitude signals and informa- 

FIG. 50 is shown with an image of the facade having 45 tion may further be transmitted via an interconnect 419 to a 

graphical user interface 501 which ofifers a full body mas- response mechanism 4110 of a graphical user interface 

sage for $45 and a push button mechanism for setting an generator 4111. A composite image comprised of optically 

appointment, graphical user interface 502 acknowledges the acquired information, computer generated image infonna- 

prior dioice ajod may offer other pleasantries or instructions. tion and finally a graphical user interface is transmitted 

FIG. 51 shows how a graphical user interface 511 might be 50 along communication line 4112 to the di^lay where the 

presented in response to activating the push button 503. The composite image is displayed in the display field, 

information associated with sub-fields 512 is dynamic with While one. will surely appreciate that, standing alone, a 

' the scheduling of the massage parlor's bookings and is graphical liser interfai^ is' a: device;"it ca^ be 

expected to change from time-to-time as others make beneficial to envisage the combination of a computer vision 

appointments. The computer of the broadcasting facility 55 system with a graphical user interface as a single device, 

responds to bookings by closing time periods as they are This is due to the fact that elements of the computer vision 

booked and offering only available appointments. In this system are intricately related to elements of the graphical 

way, user's of computer vision systems who later approach user interface. In fact, it is difficult to say they are separate 

the facade will have their gr^hical user interfaces respond as they share some elements. For example, the pointing 

appropriately. This illustrates clearly that data presented in 60 direction of the camera can serve as a pointer of the 

graphical user interfaces of computer vision systems may be graphical user interface for ^'point-and-click" operations, 

responsive to broadcasting systems. In addition, these broad- The graphical user interface field region is necessarily a 

casting S3rstems may respond to transmissions from com- subset of and is coplanar with the di^lay's planar image 

puter vision systems. field. Since the connection between computer vision systems 

FIG. 52 shows a graphical user interface displaying the 65 and graphical user interfaces is an intimate one, one might 

result of choosing a sub-field 522 associated with a time slot present the combination as a single device. Accordingly the 

4pm-5pm. This action causes an additional graphical user invention provides: 
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an apparatus iacluding Ihe combination of a computer vision a display, 

system with a graphical user interface comprised of: said computer being electronically coupled to said 

a camera; display, graphics generator, and attitude determining 

, means, 

a display; . . 

. , , . . 5 said graphics generator being responsive to said attitude 

an attitude determinmg means; ^^^^^ determining means. 

a position determining means; 2, An apparatus of claim 1, further comprising a switch 

a computer, and coupled to said computer, said switch being operable for 

a eranhical user interface allowing a user to select a position and attitude measurement 

. . . ' . , . , . 10 ^ instant in time, 
said camera havmg an optical axis and an image plane 3 ^ g^phical user interface for a computer system, the 
whereby an image of a scene being addressed by the computer system having a computer; position and attitude 
computer vision system is foraied m the unage plane determining means; and a display, said graphical user inter- 
when the optical axis is pointed into the direction of the being comprised of' 

15 a field region; and 

said display having a planar image field with a nonnal ^ periphery 

duiection as^xnated therewith, the normal direction g^^^ ^^^^ ^. ^^^^^^ 

being ahgned with the optical axis of ^d camera, or ^ j g^,^ ^^^j^ ^ j^j^ 

alternatively ahgned with an angular offset; displaying image and graphical information while a 

said attitude determming means havmg a reference direc- 20 scene is being addressed by said computer system, 

tion oolinear with the optical axis of said camera; 4. a graphical user interface of claim 3, additionally 

said position determining means having a reference point comprising a le^onse mechanism, said response mecha- 

coUocated with the intersection of the planar image nism being in communication with said field region whereby 

field and the reference direction of the attitude deter- said response mechanism operates to change the information 

mining means; 25 content of the graphical user interface. 

said computer being electronically coupled to said 5. A graphical user interface of claim 4, said response 

camera, said di;^lay, said attitude determining means, mechanism electronically connected to said position and 

and said position determining means; attitude means whereby position and attitude information is 

said graphical user interface having a field region and a transmitted to said re^nse mechanism and the content of 

periphery bounding the field region, the field region ^ graphical user mterface is responsive to position and 

being a fractional portion of the planar image field of ^^^J"^^ determmations. 

the display gra|)hical user mterface of claim 4, said graphical 

Tlie relationship between the graphical user interface and "^l^"^^ additionally oompriang a sub-field region, 

elements of the computer vision system can be fiirther ^^J^ bemg re^onsive to a cursor 

defined as follows* graphical user mterface of daim 6, said cursor bemg 

An apparatus described above, additionally comprising: ^^j^^^ corresponding to a system pointing direction. 

, „ , , . J . A graphical user mterface of claim 4, compnsmg a 

a sub-held region; a cursory and a switch, . r iJ^ • • « .u r *u * 

^ ' • ' sub-field region responsive to the position of the computer 

said sub-field region being a fractional portion of said vision system." 

field region, the sub-field having a periphery and an 40 9. A graphic user interface of claim 8, comprising pointing 

interior area, indicia with a reference point. 

said cursor being indicia movable about the planar image 10. A graphic user interface of claim 9, comprising a 

field of the di^lay including the field region and sub-field region responsive to the location of the reference 

sub-field region of the graphical user interface, point with respect to an image being displayed. 

said switch having a closed condition and an open con- ^5 11. A gr^hical user interface of claim 3, said periphery 

dition being in electronic communication with said corresponding in shape to a three dimensional object in the 

computer, operable for activating an action when the scene as it appears in perspective from the point-of-view of 

switch is simultaneously in the closed condition while computer system. 

said cursor is collocated with the interior area of the 12. A graphical user interface of claim 3, further having 

sub-field in the display field, thereby enabling the 50 indicia, for example a color highlighted region responsive to 

computer to be responsive to "point-aiKl-click" opera- location of the reference point with re^ct to the image 

tions. .-. of the scene being addressed. . 

'Although' the ^pre^rit invention has been described in - " 13. A graphical user interface of claim 3;*said field region 

considerabledetail with clear and concise language and with contains a plurality of sub-fields each contained entirely 

reference to certain preferred versions thereof including the 55 within the periphery of said graphical user interface, 

best mode anticipated by the inventor, other versions are 14. A graphical user interface of claim 13, said plurality 

possible. Therefore, the spirit and scope of the appended of sub-fields comprises at least one sub-field which corre- 

claims should not be hmited by the description of the sponds in shape to a three dimensional object in the scene 

preferred versions contained therein. being addressed as it appears from the point-of-view of the 

What is claimed is: 60 computer system. 

1. An apparatus for generating graphical user interfaces, 15. A graphical user interface of claim 3, comprising 

the apparatus comprising: indicia in the shape of a polygon having at least three sides, 

an attitude determining means; °f P^^yS^"^ corre^nding to a side of the 

periphery of the field region or the boundary of the display 

a position delermuung means; „ p^^p^^^ ^^^^ corresponding to a 

a computer; physical feature of the scene being addressed as it appears 

a graphics generator, and from the point-of-view of the computer system. 
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16. A method of di^laying a graphical user interface in a 
computer system comprising the steps: 

addressing a scene with a computer system; 
determining the position and attitude of the computer 
system; 

generating a graphical user interface in accordance with 

said position and attimde determination; and 
di^laying the graphical user interface as an image. 

17. A method of claim 16, the graphical user interface 
being dynamically movable with respect to the di^lay field. 

18. A method of claim 17, comprising the steps: 
engaging the graphical user interface with a cursor; 
activating a switch; 

dragging the graphical user interface across the display 

field while holding the switch active; and 
releasing the switch. 

19. A method of claim 18, said releasing the switch causes 
the graphical user interface to be locked to a point in the 
display field whereby the graphical user interface is associ- 
ated with a position corresponding to a point in a displayed 
image. 

20. A method of claim 18, said releasing the switch causes 
the graphical user interface to be locked to a direction in the 
display field whereby the graphical user interface is associ- 
ated with a direction of the compass. 

21. A method of claim 16, the 'generating a graphical user 
interface' step being further defined as providing a graphical 
user interface having associated therewith a plane and 
normal direction, associating the normal direction with a 
particular direction in space and displaying the graphical 
user interface in perspective with respect to the point-of- 
view of the computer. 

22. A method of claim 17, where 'dynamically movable' 
refers to movable about a reference point and is realized 
executing the following steps: 

locking a graphical user interface point to a point in an 
' image; 

engaging the graphical user interface with a cursor 
whereby the cursor is caused to become co-located with 
the interior region of the graphical user interface; 

activating a switch which causes the cursor to become 
locked to the field region of the graphical user inter- 
face; 
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advancing the cursor and consequently the field region 

about a curve; and 
deactivating the switch causing the cursor to be released 
firom the field region. 
5 23. A method of claim 17, where 'dynamically movable' 
refers to movable about a di^lay field and is realized 
executing the following steps: 

engaging the graphical user interface with a cursor to 
cause the cursor to become co-located with the interior 
area of the graphical user interface; 
activating a switch to cause the cursor to become lodced 

to the field region of the graphical user interface; 
advancing the cursor and consequently the field region in 
j5 any direction in the plane of the di^lay field but 
without limits of a preselected area; and 
deactivating the switch causing the cursor to become 

released from the field region. 
24. A method of claim 17, where 'dynamically movable' 
20 refers to movable about a di^lay field and is realized 
executing the following steps: 

tracking an object moving in a scene; 
maintaining the position of the graphical user interface 
with re^ct to the moving object. 
25 25. A method of claim I7, where 'dynamically movable' 
refers to movable with respect to the attitude of the computer 
system whereby the graphical user interface is displayed on 
a direction defined by the half-angle between the attitude of 
the computer system and a reference direction. 
^ 26. A method of claim 17, where 'dynamically movable* 
refeis to movable within a plane in the scene being 
addressed which ts movable within a plane in space such that 
panning of a reference direction causes the field of the 
graphical user interface to advance in the plane such that it 
35 has an angular appearance as the pan angle approaches 90 
degrees. 

27. A method of claim 16, said periphery is expanded in 
size such that it increases in size while the image behind 
remains the same size. 
^ 28. A method of claim 16, said periphery remains constant 
providing for an apparent size, 

29. A method of claim 16, said perq>hery being defined by 
a click-and-drag process. 

***** 



